gh-153056: Fix a data race compiling the string.Template pattern in free-threading builds by tonghuaroot · Pull Request #153057 · python/cpython

tonghuaroot · 2026-07-05T02:41:26Z

Fixes the free-threading data race in the linked issue. Template._compile_pattern() could read a pattern that another thread had just compiled and cached, then hand that already-compiled re.Pattern to re.compile(pattern, cls.flags | re.VERBOSE), which rejects a flags argument on a compiled pattern and raises a spurious ValueError. The fix returns the already-compiled pattern instead, which is idempotent and lock-free.

Testing

Lib/test/test_free_threading/test_string_template_race.py races the lazy compile across many threads with threading_helper.run_concurrently (barrier-synchronized). It uses a throwaway subclass re-armed to the uncompiled sentinel each round, so the shared string.Template class is never mutated. Without the fix, the first round raises the ValueError on many of the racing threads; with it, 0.
Lib/test/test_string/test_string.py::TestTemplate::test_precompiled_pattern is a deterministic, non-threaded regression that runs on every build: a subclass supplying an already-compiled pattern raised the ValueError at class definition before the fix and works after it.

Notes

This also lets a subclass assign an already-compiled re.Pattern to pattern, where it previously raised at class definition. The commit message calls this out as a deliberate consequence rather than leaving it implicit.
There is a separate, benign race on cls.flags (two threads may both store re.IGNORECASE); it does not affect correctness and is intentionally left out of scope to keep this change single-purpose.
I have not applied a backport label; this likely wants a 3.14 backport, but I will leave that call to a maintainer.

Issue: string.Template: data race compiling the pattern on the free-threaded build raises a spurious ValueError #153056

…n in free-threading builds Template compiles its substitution pattern lazily and caches it on the class. On the free-threaded build two concurrent first uses could race: a thread that observed the pattern another thread had just compiled would try to recompile it, and re.compile() rejects flags on an already-compiled pattern, raising a spurious ValueError. Return the already-compiled pattern instead. As a side effect, a subclass that supplies an already-compiled pattern now works too; previously it raised the same ValueError at class definition.

picnixz · 2026-07-05T07:50:03Z

I am not very fond of that but I don't see a better way of doing it except by allowing flags on a compiled pattern assuming those flags are the same. Are the flags retained by a pattern object or not? if they are not, I don't see another solution than the proposed one.

warsaw · 2026-07-07T00:54:04Z

It's actually a little more complicated because of what the docs actually say:

Alternatively, you can provide the entire regular expression pattern by overriding the class attribute pattern. If you do this, the value must be a regular expression object with four named capturing groups.

"regular expression object" implies a compiled regular expression, but that actually breaks in exactly the way you've identified in your "non-free-threading" comments and test. So the very thing that the docs tell you to do is unsupported, and I believe nothing actually tests that; all the tests set pattern to a regular expression pattern not a compiled object. Supporting a string pattern was the original supported behavior so at some point the docs got out of date.

So your fix is correct on that basis alone. Bonus that it fixes the FT race condition!

I'll take a closer look at the diff exactly, but since you're here, would you mind adding an update to the docs to describe that pattern can be a string pattern or a compiled regular expression object? With tests of the latter explicitly added, and your fix, I think it's worth backporting to 3.14 and 3.13.

warsaw

Thanks for finding and fixing this! As mentioned, I believe this is a legitimate bug regardless of the FT race, so it's good to fix and back port. Please do add a doc fix and ping me for a re-review when you're ready, as the bot will instruct.

bedevere-app · 2026-07-07T01:03:19Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

tonghuaroot · 2026-07-07T02:46:47Z

Updated the docs for pattern to state it accepts a string or a compiled regex. The compiled-object case is covered by test_precompiled_pattern. Backport labels left to you.

I have made the requested changes; please review again

bedevere-app · 2026-07-07T02:46:52Z

Thanks for making the requested changes!

@warsaw: please review the changes made to this pull request.

…fix in NEWS

tonghuaroot · 2026-07-07T02:50:56Z

Also addressed the inline comments: added the comment on the three states of pattern, and reworded the NEWS to lead with the documented-behavior fix.

I have made the requested changes; please review again

bedevere-app · 2026-07-07T02:51:01Z

Thanks for making the requested changes!

@warsaw: please review the changes made to this pull request.

read-the-docs-community · 2026-07-07T02:58:27Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #33487788 | 📁 Comparing 09bf724 against main (35c6779)

🔍 Preview build

48 files changed · ± 48 modified

± Modified

warsaw

I made one small wording suggestion, but otherwise LGTM! I'm going to update the branch and add the backport tags.

miss-islington-app · 2026-07-08T00:40:58Z

Thanks @tonghuaroot for the PR, and @warsaw for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

bedevere-app · 2026-07-08T00:41:08Z

GH-153302 is a backport of this pull request to the 3.15 branch.

miss-islington-app · 2026-07-08T00:41:12Z

Sorry, @tonghuaroot and @warsaw, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 45729033bff28f8abc36c42e802cb2853c205737 3.13

bedevere-app · 2026-07-08T00:41:15Z

GH-153303 is a backport of this pull request to the 3.14 branch.

…rn in free-threading builds (GH-153057) (#153302) gh-153056: Fix a data race compiling the string.Template pattern in free-threading builds (GH-153057) * gh-153056: Fix a data race compiling the string.Template pattern in free-threading builds Template compiles its substitution pattern lazily and caches it on the class. On the free-threaded build two concurrent first uses could race: a thread that observed the pattern another thread had just compiled would try to recompile it, and re.compile() rejects flags on an already-compiled pattern, raising a spurious ValueError. Return the already-compiled pattern instead. As a side effect, a subclass that supplies an already-compiled pattern now works too; previously it raised the same ValueError at class definition. * Trim test comments and NEWS wording * Document that the pattern attribute accepts a string or a compiled regex * Comment the three states of pattern and note the documented-behavior fix in NEWS * Update Doc/library/string.rst --------- (cherry picked from commit 4572903) Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> Co-authored-by: Barry Warsaw <barry@python.org>

tonghuaroot · 2026-07-08T01:53:11Z

Thanks. I looked into the 3.13 backport and I do not think it is warranted.

On 3.13, Template compiles its pattern eagerly in __init_subclass__ at class definition time, so there is no lazy first-use compile-and-cache step for concurrent threads to race on. The lazy _compile_pattern / _TemplatePattern mechanism that this PR guards was introduced in 3.14, so the data race does not exist on 3.13 (the free-threading test here also relies on string._TemplatePattern, which 3.13 does not have).

The only 3.13-relevant part would be the secondary fix, where a subclass that supplies an already-compiled pattern raises a spurious ValueError. That is a minor pre-existing issue rather than the race, and I would leave it out of scope here. I suggest dropping the needs backport to 3.13 label.

…rn in free-threading builds (GH-153057) (#153303) gh-153056: Fix a data race compiling the string.Template pattern in free-threading builds (GH-153057) * gh-153056: Fix a data race compiling the string.Template pattern in free-threading builds Template compiles its substitution pattern lazily and caches it on the class. On the free-threaded build two concurrent first uses could race: a thread that observed the pattern another thread had just compiled would try to recompile it, and re.compile() rejects flags on an already-compiled pattern, raising a spurious ValueError. Return the already-compiled pattern instead. As a side effect, a subclass that supplies an already-compiled pattern now works too; previously it raised the same ValueError at class definition. * Trim test comments and NEWS wording * Document that the pattern attribute accepts a string or a compiled regex * Comment the three states of pattern and note the documented-behavior fix in NEWS * Update Doc/library/string.rst --------- (cherry picked from commit 4572903) Co-authored-by: tonghuaroot (童话) <tonghuaroot@gmail.com> Co-authored-by: Barry Warsaw <barry@python.org>

warsaw · 2026-07-08T22:15:23Z

The only 3.13-relevant part would be the secondary fix, where a subclass that supplies an already-compiled pattern raises a spurious ValueError. That is a minor pre-existing issue rather than the race, and I would leave it out of scope here. I suggest dropping the needs backport to 3.13 label.

Thanks for looking into this. I think it's still worth backporting the regex object support fix and it wasn't that hard to cherry pick and resolve.

bedevere-app Bot added the awaiting review label Jul 5, 2026

bedevere-app Bot mentioned this pull request Jul 5, 2026

string.Template: data race compiling the pattern on the free-threaded build raises a spurious ValueError #153056

Open

Trim test comments and NEWS wording

21fd115

warsaw requested changes Jul 7, 2026

View reviewed changes

Comment thread Lib/string/__init__.py

Comment thread Lib/test/test_string/test_string.py

Comment thread Misc/NEWS.d/next/Library/2026-07-05-12-00-00.gh-issue-153056.tMpLat.rst Outdated

bedevere-app Bot removed the awaiting review label Jul 7, 2026

bedevere-app Bot added the awaiting changes label Jul 7, 2026

warsaw self-assigned this Jul 7, 2026

Document that the pattern attribute accepts a string or a compiled regex

9518fcb

bedevere-app Bot added awaiting change review and removed awaiting changes labels Jul 7, 2026

bedevere-app Bot requested a review from warsaw July 7, 2026 02:46

Comment the three states of pattern and note the documented-behavior …

764fa3f

…fix in NEWS

warsaw approved these changes Jul 8, 2026

View reviewed changes

Comment thread Doc/library/string.rst Outdated

bedevere-app Bot added awaiting merge and removed awaiting change review labels Jul 8, 2026

warsaw added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes needs backport to 3.15 pre-release feature fixes, bugs and security fixes labels Jul 8, 2026

warsaw added 2 commits July 7, 2026 17:07

Merge branch 'main' into template-compile-ft-race

ab17838

Update Doc/library/string.rst

09bf724

warsaw enabled auto-merge (squash) July 8, 2026 00:08

warsaw merged commit 4572903 into python:main Jul 8, 2026
50 of 51 checks passed

bedevere-app Bot removed the awaiting merge label Jul 8, 2026

bedevere-app Bot removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 8, 2026

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Jul 8, 2026

Uh oh!

Uh oh!

Conversation

tonghuaroot commented Jul 5, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Notes

Uh oh!

picnixz commented Jul 5, 2026

Uh oh!

warsaw commented Jul 7, 2026

Uh oh!

warsaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bedevere-app Bot commented Jul 7, 2026

Uh oh!

tonghuaroot commented Jul 7, 2026

Uh oh!

bedevere-app Bot commented Jul 7, 2026

Uh oh!

tonghuaroot commented Jul 7, 2026

Uh oh!

bedevere-app Bot commented Jul 7, 2026

Uh oh!

read-the-docs-community Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

warsaw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

miss-islington-app Bot commented Jul 8, 2026

Uh oh!

bedevere-app Bot commented Jul 8, 2026

Uh oh!

miss-islington-app Bot commented Jul 8, 2026

Uh oh!

bedevere-app Bot commented Jul 8, 2026

Uh oh!

tonghuaroot commented Jul 8, 2026

Uh oh!

warsaw commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tonghuaroot commented Jul 5, 2026 •

edited by bedevere-app Bot

Loading

read-the-docs-community Bot commented Jul 7, 2026 •

edited

Loading